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ABSTRACT 


Systems Engineering and Integration (SE&I) is a critical discipline for developing new space systems. In 2005, 
NASA performed an internal study of 24 NASA and Department of Defense (DoD) programs to evaluate methods 
of integrating SE&I practices and determine their effectiveness. The goal of the study was to determine the best 
SE&I implementation strategy for the Ares Projects. The study identified six SE&I organizational structures, 
encompassing an array of relationships and levels of responsibility between government agencies and other 
participating parties. The organizational structure used most often was using a prime contractor with government 
SE&I responsibility and government technical insight. However, data analyses did not establish a positive 
relationship between program success or development costs and specific SE&I organizational types. The study 
determined that large, long-duration, complex programs reach their technical goals, but rarely meet schedule or cost 
goals. Programs have failed or been terminated due to lack of technical insight, relaxing of SE&I processes, and 
unstable external factors. While any organizational structure can be made to work, success is more likely with fewer 
program complexities. This study was instrumental in helping Ares select an organizational structure following the 
same SE&I and oversight process used during the Apollo program. 
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Introduction 


Systems Engineering and Integration (SE&I) is a 
critical discipline for developing new space systems. 
SE&I grew out of the need to ensure cost and 
technical performance in U.S. ballistic missile and 
space launch systems in the 1950s and 1960s: 

Scientists and engineers found that they needed 
some individuals to coordinate the information 
flowing among working groups. These ‘systems 
engineers’ created and maintained documents 
that reflected the current design, and they 
coordinated design changes with all those 
involved in the program. 1 

For organizations to learn, adapt, and sustain 
adaptations, they must have processes that are both 
flexible and durable." Therefore, the Ares Projects’ 
management team wanted an organizational structure 
that met those needs as well as the overarching 
requirements of the U.S. Space Exploration Policy. 
These requirements include replacing the Space 
Shuttle’s crew and cargo-carrying capabilities to the 
International Space Station (ISS) and building launch 
vehicles capable of transporting crew and cargo to 
the Moon and beyond. 


In 2005, NASA performed an internal study of 24 
NASA and Department of Defense (DoD) programs 
to evaluate the effectiveness of various methods of 
integrating SE&I practices. The goal of the study was 
to determine the best SE&l implementation strategy 
for the Ares Projects. 

1.0 Study Overview and Requirements 

The SE&I study overview included a description of 
Ares’ basic SE&I requirements, study ground rules 
and assumptions, methodology, program/project 
selection rationale, a template used for data 
collection, and a description of program types 
included in the study. 

To ensure proper SE&I implementation, the study 
mandated the following requirements: 

• The government has ultimate responsibility 
for the Project’s outcome. 

• The Project must have a Systems Integrator 
(either government or contractor). 

• There must be adequate competencies/ 
resources available to perform Project tasks. 
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The third point was of particular concern for the Ares 
team, as NASA would need to rebuild its internal 
competencies in SE&I and vehicle design. 

1 . 1 Study Ground Rules and Assumptions 

The study’s ground rules and assumptions provided a 
common baseline for the options considered. These 
ground rules and assumptions included: 

• References for actual cost trending came 
from the Resource Data Storage and 
Retrieval (REDSTAR) database, which 
served as the basis for the recent NASA cost 
growth study. DoD cost estimates were 
extracted from General Accounting Office 
(GAO) reports and other public news 
sources. The data to support the study were 
derived from public websites and personal 
interviews (lessons learned). To the extent 
possible, more than two personal interviews 
were conducted for each program/project. 
Interviewees represented various 
program/project disciplines. 

• Programs/projects were considered 
“technically successful” if the primary 
mission objectives were met. Similarly, a 
program/project was considered a “failure” 
if the primary mission objectives were not 
met. 

• Terminated programs/projects refer to 
programs/projects that were terminated 
during development. 

1.2 Study Methodology 

The study used the following methodology to analyze 
SE&I organizational structures and implementation, 
identify common trends across DoD and NASA 
programs/ projects, and identify independent factors 
influencing program/project outcomes: 

• Identified SE&I organizational 
considerations. 

• Defined metrics. 

• Identified NASA/DoD programs/projects for 
the study. 

• Completed a program/project template. 

• Collected data/conducted interviews. 

• Reviewed independent reports/GAO reports 
for lessons learned/recommendations. 

• Identified factors that influenced SE&I 
implementation across programs/projects. 


• Assessed data and identified “best practices” 
and lessons learned across programs/projects 
related to SE&I implementation. 

2,0 Columbia and the Need for Improved Systems 

Engineering Practices 

The SE&I study included a review of the 
recommendations from the Columbia Accident 
Investigation Board (CAIB) Report 111 and the Diaz 
Commission, which was formed soon after CAIB. 

The Diaz Commission was chartered to analyze the 
implications of the CAIB findings for other NASA 
programs. SE&I-related CAIB findings and 
recommendations are summarized below: 

• “F7.4-3: Over the last two decades, little to 
no progress has been made toward attaining 
integrated, independent, and detailed 
analyses of risk to the Space Shuttle system. 

• F7.4-4: System safety engineering and 
management is separated from mainstream 
engineering, is not vigorous enough to have 
an impact on system design, and is hidden in 
the other safety disciplines at NASA F1Q. 

• F7.4-5: Risk information and data from 
hazard analyses are not communicated 
effectively to the risk assessment and 
mission assurance processes. The [CAIB] 
Board could not find adequate application of 
a process, database, or metric analysis tool 
that took an integrated, systemic view of the 
entire Space Shuttle system. 

• F7.4-6: The Space Shuttle Systems 
Integration Office handles all Shuttle 
systems except the Orbiter. Therefore, it is 
not a true integration office. 

• R7.5-1: Establish an independent Technical 
Engineering Authority that is responsible for 
technical requirements and all waivers to 
them, and will build a disciplined, systemic 
approach to identifying, analyzing, and 
controlling hazards throughout the life cycle 
of the Shuttle System. The independent 
technical authority does the following as a 
minimum: 

- Develop and maintain technical 
standards for all Space Shuttle Program 
projects and elements; 

- Be the sole waiver-granting authority 
for all technical standards; 

- Conduct trend and risk analysis at the 
subsystem, system, and enterprise 
levels; 
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- Own the failure mode, effects analysis, 
and hazard reporting systems; 

- Conduct integrated hazard analysis; 

- Decide what is and is not an anomalous 
event; 

- Independently verify launch readiness; 
and 

- Approve the provisions of the re- 
certification program called for in 
Recommendation. 

• R9.1-1: The Technical Engineering Authority 
should be funded directly from NASA HQ, and 
should have no connection to or responsibility 
for schedule or program cost. 

• R7.5-2: NASA HQ Office of Safety and Mission 
Assurance should have direct line authority over 
the entire Space Shuttle Program safety 
organization and should be independently 
resourced. 

• R7.5-3: Reorganize the Space Shuttle Integration 
Office to make it capable of integrating all 
elements of the Space Shuttle Program, 
including the Orbiter. 

• R9.1-1 : Prepare a detailed plan for defining, 
establishing, transitioning, and implementing an 
independent Technical Engineering Authority, 
independent safety program, and a reorganized 
Space Shuttle Integration Office as described in 
R7.5-1, R7.5-2, and R7.5-3. In addition, NASA 
should submit annual reports to Congress, as 
apart of the budget review process, on its 
implementation activities.” 

Finally, the CAIB Report (p. 179) stated: “As a result 
[of the Space Flight Operations contract], 
experienced engineers changed jobs, NASA grew 
dependent on contractors for technical support, 
contract monitoring increased, and positions were 
subsequently staffed by less experienced engineers 
who were placed in management roles. Collectively, 
this eroded NASA’s in-house engineering and 
technical capabilities.” 

The CAIB report and the Diaz Commission both 
addressed common themes, as depicted in Figure 1. 
The most important finding is that NASA needs to 
rebuild its SE&I leadership and technical capabilities. 

This study was designed and executed with the 
results of the CAIB and the Diaz panels in mind. The 
goal was to give the agency information residting in 
an organization capable of addressing the problems 
noted in these reports. 


CAIB Observations Diaz Roport Agency-Wide Thornes 


Management practices 
are a cause of the 
accident 


Leaders must lead by example, creating conditions and a cullive 

for safety and mission success 

Leaders must balance schedule and nsk 

Leaders should allow and encourage diversity of views, efenlnate 

retribution towards those with differing opinions, and understand 

that ‘No' Is an acceptable answer 

Leaders should be grown throughout al levels of the organization 
through succession planning and developmental experiences 


NASA has not 
demonstrated the 
characteristics of a 
learning organization 


• NASA should provide robust simulation and emergency response 
— ^ trairang 

N * * Tools, databases, and models should be developed and used 
appropriately 


Communication did not 
flow effectively 


-A ■ 
-V . 


Anomalies should be considered problems unless proven otherwise 
Responsibility . authority . and accountability must be cleaity 
understood and communicated 

Communications noed to flow both up and down the chain of 
command 

Dtvorse viewpoints must be fostered and minority views 
considered 


NASA has not followed 
As own rules 


NASAs in-house 
capacities & expertise 
have eroded 


-a ' 
-v : 


fN 4 
V . 


Communication practices must be v aklatod and tested to ensure 
of foctlvo commurxc.it ion across NASA 


Requirements, policies, procedures, and directives must be 
examined and adhered to 

Best practices and lessons learned must be incorporated 
The entire woikforce must be aware of and understand the rules 


Analytical models and sanitation toots must be used appropriately 
NASA must address the loss of techncal expertise due to 
retirement and outsourcag 
Advanced technical capabilities must be developed 


NASAs safety and 
engmeenng organizations 
lack authority and 
Independence 


• NASA must ensure systematic checks and balances are m place 
i • NASA must implement independent safety and technical 
, ’ organizations 

~y • Roles and responsibilities must be clearly defined and w*doty 
understood 


NASA fails to account for 
program nsk 


r 

-Y 


NASA must establish a consistent set of nsk assessment tools, 
appbed m a uniform way . across ail programs 
NASA management must have a clear understanding of safely 
requroments and risks 
Associated wdh key decisions 

NASA must stop accepting increasing levels of nsk without 
understanding the total program nsk 


Figure 1. CAIB and Diaz Commission observations lv 
clearly mapped NASA's SE&I challenges. 


3.0 Program/Proiect Types Included in the Study 

The study examined the SE&I implementation 
capabilities of 24 NASA/DoD programs/projects. 
These programs/projects were selected to provide the 
greatest range of characteristics and to represent both 
past and current SE&I implementation efforts. The 
following factors were considered in determining 
which programs/projects to study: 

• Small projects to large programs. 

• Successful versus failed/terminated 
programs/proj ects . 

• Completed and ongoing development. 

• Data availability (based on study time 
constraints). 

Table 1 provides a summary of the program/project 
types, the associated development times, and current 
program/project phases. Failed or terminated 
programs are highlighted in red. 
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Table 1 . Selected Program/Project Type Summary. 


Program Type 

Duration (DDT&E) 
(Years) 

Current Phase 

Program 1 

2.5 

Operatonal 

Program 2 

3 

Mission Failure 

Program 3 

3 

Mission Failure 

Program 4 

3 

Operatonal 

Program 5 

3 

M ission completion 

Program 6 

5 

"erminated 

Program 7 

5 

"erminated 

Program 8 

6 

Integration & Test 

Program 9 

6 

Operatonal 

Program 10 

7 

Operatonal 

Program 1 1 

7 

Operatonal 

p rogram 12 

8 

Mission Failure 

Program 13 

6 

Development 

Program 14 

8 

Development 

Program 15 

6 

Operatonal 

Program 10 

9 

Operatonal 

Program 17 

9 

Operatonal 

Program 18 

9 

Operatonal 

Program 19 

11 

M ission Completion 

Program 20 

11 

Development 

Program 21 

11 

Operatonal 

Program 22 

12 

Development 

Program 23 

23 

Terminated 

Program 24 

24 

Operatonal 


In evaluating these programs/projects, the study 
identified six primary SE&I organizational structures: 

1 . Lead systems integrator (LSI) with SE&I 
responsibility and government technical 
insight. 

2a. Integration contractor with government 
SE&I responsibility (government insight). 

2b. Integration contractor with government 

SE&I responsibility (government oversight). 

3a. Prime contractor with SE&I responsibility 
(government insight). 

3b. Prime contractor with SE&I responsibility 
(government oversight). 

3c. Prime contractor with SE&I responsibility 
(govemment/industry partnership). 

4a. Prime contractor with government SE&I 
responsibility (government insight). 

4b. Prime contractor with government SE&I 
responsibility (government oversight). 

4d. Prime contractors with total system 
performance responsibility (TSPR). 

5. Prime contractor with government SE&I 
responsibility and integration products 
through a Federally Funded Research and 
Development Center (FFRDC). 

6. Government/FFRDC in-house development 
with SE&I responsibility and function. 

Detailed descriptions of these program/project types 
are provided below. However, the following terms 
will first be defined for clarity and consistency. 

• Oversight: Using government resources to 
monitor all aspects of contractor 
performance in product delivery and 
execution of approved processes. 


• Insight: A customer’s risk-based 
understanding, validation, and surveillance 
of a supplier’s management systems and 
process performance metrics to assure 
product quality and contract compliance. 

• Conventional Prime Contractor: Develops 
and builds what it can and subcontracts out 
what it cannot. A prime contractor provides 
an end item to the government. 

• Lead Systems Integrator (LSI): Plans and 
performs Program Management (PM), 
Systems Engineering (SE), Systems 
Integration (SI), and acquisition. 

• System-of-Systems (SoS): A set or 
arrangement of interdependent systems that 
are related or connected to provide a given 
capability. The loss of any part of the system 
will degrade the performance or capabilities 
of the whole. An example of an SoS could 
be interdependent information systems. 
While individual systems within the SoS 
may be developed to satisfy the peculiar 
needs of a given user group (such as a 
specific service or agency), the information 
they share is so important that the loss of a 
single system may deprive other systems of 
the data needed to achieve even minimal 
capabilities. 

• Total System Performance Responsibility 
(TSPR): Under a TSPR contract, the 
contractor assumes responsibility for system 
life-cycle management. The anticipated 
benefits of a TSPR arrangement include 
decreased product delivery time, reduced 
costs and data, reduced program office 
manpower, fewer Engineering Change 
Proposals (ECP), reduced total ownership 
cost, and increased quality and readiness. 
(TSPR is also referred to as Full Services 
Contracting.) The government does not 
typically provide the traditional level of 
government oversight with this type of 
contract. 

The six primary types of organizational relationships 
listed previously are detailed below and depicted in 
block diagram form in Figure 2. 

1. LSI with SE&I responsibility. 

• Little/no hardware/soft ware development 
responsibility for the LSI. 

• LSI issues most/all subcontracts. 

• Responsible for total systems performance. 
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2a. Integration contractor with government SE&I 
responsibility (may also be a prime contractor, with a 
separate integration task/contract). 

• Integration contractor performs SE&I 
functions across multiple primes. 

• Government-issued prime contracts. 

• Government insight. 

2b. Integration contractor with government SE&I 
responsibility (may also be a prime contractor, with a 
separate integration task/contract). 

• Integration contractor performs SE&I 
functions across multiple primes. 

• Government-issued prime contracts. 

• Government oversight. 

3a. Prime contractor with SE&I responsibility. 

• Issues all subcontracts. 

• Government insight. 

3b. Prime contractor with SE&I responsibility. 

• Issues all subcontracts. 

• Government oversight. 

3 c. Prime contractor with SE&I responsibility 
(government/industry partnership). 

• Issues all subcontracts. 

• Government insight. 

4a. Prime contractor with government SE&I 
responsibility. 

• Both government and prime contractor issue 
contracts. 

• Government insight. 

4b. Prime contractor with government SE&I 
responsibility. 

• Both government and prime contractor issue 
contracts. 

• Government oversight. 

4d. Prime contractor with TSPR. 

• Both government and prime contractor issue 
contracts. 

• Government insight. 

5. Prime Contractor with government SE&I 
responsibility and SE&I products (through FFRDC/ 
Systems Engineering Technical Assistants (SETA)). 

• Performs integration analysis, using 
contractors/SE Technical Assistants (SETA) 
or FFRDCs perform SE&I function. 

6. Government (FFRDC) in-house development with 
SE&I responsibility and SE&I function. 

• Government and FFRDC issue contracts. 



Figure 2. The block diagrams depict the operating 
relationships between government and its partners 
for the various program types. 


3.1 Program Characteristics/SE&l 

Organizational Structure Relationships 

The six SE&I organizational types each have key 
program characteristics associated with them. 
Program characteristic variables include: cost, 
development time, production rates, architecture type 
(modular, integrated), international partners, 
operational environment (geosynchronous, deep 
space, etc.), number of development locations 
(government only), number of critical technologies, 
and new infrastructure requirements. 

• Type 1: Similar characteristics include net- 
centric, tightly integrated architectures, long 
development phases (greater than 8 years), 
large number of critical technologies, new 
infrastructure development, long operational 
phase, and development costs greater than 
$5 billion. 

• Type 2: Similar characteristics include long 
development phases (greater than 5 years), 
modular architectures, multiple govermnent 
development locations, and new 
infrastructure requirements. 

• Types 3 and 4: Types 3 and 4 represented 14 
of the 24 programs. Common Type 3 
program/project characteristics included 
modular architectures, limited production, 
and new infrastructure requirements. There 
were varying characteristics across Type 4 
SE&I organizational types. 

• Type 5: Only one type program was 
included in this study. 

• Type 6: Similar characteristics included 
short development schedules, with high 
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heritage (design/hardware) modular 
architectures. 

3.2 Advantages and Disadvantages of 
Program/Project Types 

The study identified key strengths and concerns for 
each of the program types, which are addressed 
below. 

Type 1: LSI with SE&I Responsibility 1 

In these types of programs, little or no 
hardware/software development is performed. The 
LSI issues most or all of the contracts and is 
responsible for total systems performance. 

This type of organization is typically implemented 
for large programs, with net-centric, integrated 
architectures, and multiple technology 
developments/insertions. 

Key Strengths 

• Competition during concept phase. 

• Reduced government contract 
administration. 

• Contracts administered by one source (LSI). 

• Flexibility in acquiring core competencies to 
support long-term development. 

Significant Areas to Address in Implementation 
Planning 

• Project management (PM) and technical 
expertise for government personnel are 
needed to monitor programs. 

• Program uncertainties derived from external 
factors can lead to significant change orders 
that require recurring system assessments, 
impacts to contracts, and Earned Value 
Management (EVM). 

• Firewall/proprietary data issues associated 
with competitive acquisition. 

• Addressing the multiple 
standards/processes/test philosophies 
existing across multiple government 
organizations/industry participants. 

• Handling of government legacy 
hardware/facilities/contracts and 
international partner deliverables that are not 
typically included in LSI contract (key 


Programs included in study are still in development 
stage/limited lessons learned for supporting implementation 
assessment 


feature of LSI is management of all 
contracts for supporting integrated trades, 
interface definition, and analyses, 
commonality, interoperability, integrated 
risk management, and integrated testing). 

• Clear definition of government/industry 
roles and responsibilities and decision- 
making process. 

• International partner roles, responsibilities, 
contractual structure, and participation in 
interface definition and implementation. 

• Alignment of government/industry 
organizational structures to support joint 
teaming. 

• Data rights issues and proprietary data are 
issues for modeling and simulation and for 
capturing the design to ensure that 
government captures that knowledge. 

• Impacts of staggered element development 
on the overall SE&I function. 

• Organizational conflicts of interest. 

Type 2: Integration Contractor with Government 
SE&I Responsibility 

These programs/projects perform SE&I functions 
across multiple prime contractors, where the contract 
may be issued to one of the prime contractors, as a 
separate contract/task. 

This type of organization is typically implemented 
for large programs with multiple prime contractors 
associated with multiple government locations. 

Key Strengths 

• Clear government responsibility for the 
design and development, and system 
performance. 

• Single point of contact for the SE&I 
function. 

• Flexibility in acquiring core competencies 
across program life cycle. 

• The ability of the integration contractor to 
compete for other program work is 
dependent on program characteristics and is 
determined by program management. 

Significant Areas to Address in Implementation 
Planning 

• Contractual agreements between the 
integration contractor and other primes, 
international partners, or government- 
developed equipment/facilities are necessary 
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to support integrated trades, analyses, and 
tests. 

• Contractual agreements between integration 
contractor and prime contractor, even if they 
are part of the same company. 

• Program uncertainties derived from external 
factors can lead to significant change orders, 
requiring recurring system assessments, 
impacts to contracts, and EVM. 

• Risk assessment for determining 
government technical involvement, 
independent analyses, and verification. 

• Defining clear roles and responsibilities for 
the multiple government 
organizations/industry participants. 

• Alignment of government/industry 
organizational structures to support joint 
government/industry teaming. 

• Decision authority and program control 
board process across multiple primes for 
effective interface definition and 
management. 

• International partner roles, responsibilities, 
contractual structure, and participation in 
interface definition and implementation. 

• Addressing the multiple 
standards/processes/test philosophies have 
existing across multiple government 
organizations and industry. 

• Data rights issues, proprietary data issues for 
modeling and simulation and for capturing 
design to ensure knowledge capture by the 
government. 

• Impacts of staggered element development 
on overall the SE&I function. 

• Organizational conflicts of interest. 

Type 3: Prime Contractor with SE&I 
Responsibility (includes Government/Industry 
Partnerships) 

In these types of programs/projects, the Prime 
Contractor issues all contracts. This organizational 
structure is typically implemented for large and small 
programs, with various types of architectures. 

Key Strengths 

• Reduced government administrative 
overhead. 

• Reduced government technical support. 

• Increased flexibility for the prime 
contractor. 


• All contracts managed by one prime 
contractor. 

Significant Areas to Address in Implementation 
Planning 

• Government’s visibility into 
program/project management control 
(programmatic and technical). 

• Clearly defined roles and responsibilities for 
government and industry personnel (life 
cycle). 

• Risk assessment for determining 
government technical involvement, 
independent analyses, and verification. 

• Decision-making authority definition. 

• Impacts of changes in anticipated customer 
base/market on partnership. 

• Impacts of technical challenges during 
development phase on partnership. 

• Other external factors impact on either 
prime contract/partnership. 

• Data rights issues, proprietary data issues for 
modeling and simulation and for capturing 
design to ensure knowledge capture by the 
government. 

• Impacts of staggered element development 
on overall SE&I function. 

• Organizational conflicts of interest. 

Type 4: Prime Contractor with Government 
SE&I Responsibility 

In these types of programs/projects, both government 
and industry issue and manage contracts. The 
organizational structure is implemented for both large 
and small programs. 

Key Strengths 

• Clear government SE&I responsibility. 

• Requirements allocation and balancing 
across program elements throughout 
development phase decreases the potential 
for conflict of interest. 

• Minimizes potential for organizational 
conflicts of interest. 

Significant Areas to Address in Implementation 
Planning 

• International partner roles, responsibilities, 
and participation in interface definition and 
implementation. 
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• External factor impacts on SE&I resources, 
standards/processes, and integrated testing. 

• Integration contracts between prime and 
government-managed contracts support 
timely and effective interface definition, 
analyses, issue resolution, integrated testing, 
standards, and processes. 

• Multiple reporting paths to government 
program management’s impact on 
requirements allocation, resources, etc. 

• Data rights issues, proprietary data issues for 
modeling and simulation and for capturing 
design to ensure knowledge capture by the 
government. 

• Impacts of staggered element development 
on overall SE&l function. 

Type 5: Prime Contractor with Government 
SE&I Responsibility & SE&I Function 

In this organization, government issues a prime 
contract for development and launch services. The 
organizational structure means that a government 
FFRDC provides analytical integration. This 
structure has been implemented for both large and 
small programs. 

Key Strengths 

• Clear government responsibility for 
interface product development and 
analytical integration across system. 

• Provides continuous and consistent 
independent risk-based analyses. 

• Timely interface issue identification and 
resolution. 

• Resource flexibility with FFRDC program 
support. 

• Minimizes potential for organizational 
conflicts of interest. 

Significant Areas to Address in Implementation 
Planning 

• PM and technical expertise for government 
personnel. 

• Long-term FFRDC commitment. 

• Data rights issues and proprietary data issues 
for modeling and simulation and for 
capturing design to ensure the government 
captures that knowledge. 

• Impacts of staggered element development 
on overall SE&l function. 


Type 6: Government In-House Development with 
SE&I Responsibility and Function 

In this type of program/project, both the government 
and FFRDC issue contracts. The organizational 
structure is implemented for small programs, with 
focused, critical technology developments/insertions. 

Implementation Strengths 

• Clear government (FFRDC) responsibility 
for management and integration of system. 

• Strengthening of FFRDC PM and technical 
core competencies. 

• Flexibility in selection of core competencies 
across program life cycle. 

• All contracts and SE&I managed by one 
entity, with minimal/no potential for 
conflicts of interest. 

Other Areas to Address in Implementation 
Strategy 

• Government PM and technical expertise 
government. 

• Long-term FFRDC commitment. 

• Multiple reporting paths to government 
program management impacts requirements 
allocation, resources, etc. 

4.0 Other Factors Affecting Program/Proiect Success 

Several key independent external and internal factors, 
relevant to SE&I implementation, were identified as 
part of this study, through individual program/project 
lessons learned, independent assessment reports and 
interviews. 

4.1 External Factors 

External factors included major policy/mission 
changes, funding instabilities/inadequate reserves, 
tight programmatic constraints (funding/schedule 
caps), multiple reporting management paths, and 
change in customer/contractor base. 

4.2 Internal Factors 

Internal factors were identified as lack of integration 
agreements between various contractors, lack of 
technical oversight, poor communications, multiple 
critical technologies, inadequate risk management, 
lack of rigorous processes and standards, lack of core 
competencies, lack of independent analyses, lack of 
comprehensive integrated tests, large number of 
complex interfaces, Modeling and Simulation (M&S) 



issues, international partnership technical issues, and 
lack of effective and timely integrating solutions 
across multiple organizations. 

4.3 Cross-Cutting Trends of Successful 
Programs/Projects 

Characteristics of successful programs/projects with 
minimal or no-cost overruns were evaluated as part 
of the study. Common characteristics included: 

• Funding stability/adequate reserves. 

• Experienced program management. 

• Limited critical technologies, with high 
heritage designs. 

• Strong communications (implemented 
through “badgeless” teams). 

• Application of rigorous standards/processes. 

• Less than or equal to 7 year development 
effort. 

• Successful programs had the least number of 
internal/external factors identified by the 
study. 

A survey in IEEE Engineering Management Review 
examined the “capacity for engineering systems 
thinking (CEST),” and asked professionals in the 
field their opinions of what particular abilities and 
skills were necessary for effective SE&I. Among the 
CEST capabilities were: 

• Understanding the whole system and seeing 
the big picture 

• Understanding interconnections and 
“closed-loop” thinking - i.e., a “circular” 
view of issue causality rather than straight- 
line, “linear” thinking - the ability to offer 
several possible explanations for a problem, 
not only one 

• Understanding systems synergy - i.e., the 
ability to identify new forms of integration 
or emergent properties of combined systems 

• Understanding the system from multiple 
perspectives 

• Thinking creatively 

• Understanding systems without getting stuck 
on details; tolerance for ambiguity and 
uncertainty 

• Understanding the implications of a 
proposed change 

• Understanding a new system/concept 
immediately upon presentation 

• Understanding analogies and parallelism 
between systems 


• Understanding limits to growth. 

NASA managers for the Apollo lunar program cited 
the agency’s internal expertise as a critical success 
factor for landing a man on the Moon within a 
decade: “[Research and Development Operations] 
provides technical knowledge in depth to solve the 
technical problems, but at the same time carefully 
avoiding any interference with contract 
management.”' Based on Wernher von Braun’s 
experiences at Peenemiinde and Redstone Arsenal, 
the early Apollo organization used an “arsenal” 
management style, which required a highly capable 
in-house knowledge base that sought contractor 
assistance only for production purposes. Von Braun 
emphasized a “dirty hands” approach to engineering 
among his technical team as the best preparation for 
evaluating contractor standards and proposals.' 1 

At a panel discussion honoring the 20 th anniversary 
of the Apollo 1 1 landing, NASA managers Maxime 
Faget and Christopher Kr aft echoed these sentiments 
on in-house technical competence. Faget stated, “I 
think we were ahead of the contractors. As a matter 
of fact, before we even put the [request for proposal] 
out, we pretty much knew what we wanted and stated 
it. The Apollo Command Module was designed more 
or less by our people.”' 11 Chris Kraft seconded 
Faget’ s comments, saying, “I think that what we 
recognize, from Mercury to Gemini to Apollo, was 
that we needed a certain percentage of funds to do 
these kinds of things in Max’s laboratory which gave 
our people straight, hands-on knowledge, first-hand 
knowledge, and it allowed us to build system that we 
built. . .to use in Apollo.”' 111 

These are some of the abilities that the Ares Projects 
must teach and encourage if they are to 
institutionalize systems engineering as a core NASA 
competency in the future. 

4.4 Cross-Cutting Trends of Failed/Terminated 
Programs/Projects 

Program characteristics of successful 
programs/projects were also evaluated as part of the 
study. Common characteristics included the 
following: 

• Funding instability. 

• Government insight, or insight transitioned 
to oversight during development. 

• Poorly executed systems engineering 
processes. 
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• Failed/terminated programs had a significant 
number of the external/internal factors 
present. 

4.5 Benchmarking 

Upon conclusion of this study, NASA performed a 
benchmarking study to investigate SE&I practices in 
companies and organizations outside of the aerospace 
industry. The benchmarking studies had two major 
goals: incorporate innovative practices from other 
industries and validate the results of this study. The 
initial selection criteria for study companies included: 

• Long-term, high-risk/reward programs 

• Need and ability to manage multiple 
technical and programmatic interfaces 

• Need and ability to manage multiple 
supplies and to mitigate organizations 
conflicts of interest 

• Ability to perform continuous technology 
infusion throughout the life cycle of the 
program 

Approximately 50 organizations were initially 
selected and 5 organizations were chosen for 
benchmark visits. These organizations included 
consumer products, construction and engineering 
management, telecommunications, integrated energy 
producers and retailers, and a large military 
development program. 

5.0 Conclusions 

Over a two-month period, the study team derived a 
number of key conclusions that were decisive in 
determining which type of organization structure the 
Ares Projects would eventually use. 

First, large, long-duration, technically complex 
programs/projects identified in the study that met 
high-level technical requirements have rarely met 
schedule and have far exceeded original cost 
estimates. There is evidence that all of these 
programs/projects have been significantly impacted 
by external factors, regardless of the SE&I 
organizational structure type. 

Second, NASA’s recent successes have been with 
smaller, short-duration development 
programs/projects, using heritage hardware/software, 
focused technology development, technical oversight, 
“badge-less” teaming, and stable external factors. 

Third, program/project failure/termination common 
trends include technical insight, SE&I process 
relaxation, and unstable external factors. 


Fourth, independent report findings cited review and 
oversight performance/ process, 
culture/communications, and budget/cost estimates, 
as the most common areas for improvement. 

Fifth, There is no clear optimum SE&I organizational 
structure type (i.e., no one size fits all). Both 
implementation of the various SE&I organizational 
types, and independent factors appear to influence 
program outcome. Each SE&I organizational 
structure type has its own strengths and weaknesses, 
and any SE&I organizational structure can be made 
to work given time and effort. Addressing these 
implementation findings during program planning 
can improve a program’s/ project’s probability of 
success. Programs with the least number of factors 
present during development appeared to meet 
mission objectives, with minimal cost and schedule 
overrun. Failed/terminated programs had a significant 
number of these factors present. Current programs 
with a significant number of these factors have 
already experienced cost and schedule overruns. 

Sixth, evolving system complexities and integrated 
architectures add to the complexity of implementing 
successful SE&I across programs/projects, including 
M&S, where tool ownership, implementation, and 
proprietary data issues have been identified. 

Seventh, the most common successful SE&I 
organizational structure type in this study was the 
structure Type 4b, where government maintained 
integration responsibility, with the prime contractor 
providing SE&I products and the government 
providing technical oversight. This organizational 
type has been implemented by both small and large 
programs, with varying program characteristics. 

Eighth, there was not sufficient data to establish a 
relationship between program development costs and 
a specific SE&I organizational type selection. Type 1 
programs were not implemented for the smaller 
development budgets. Similarly, Type 6 programs 
were not implemented for the larger development 
budgets. Type 2, 3, and 4 programs were observed 
for both small- and large-budget categories. 

The results of the benchmarking studies also 
validated these conclusions. Each organization 
operated under different approaches to systems 
engineering (while most did not refer to it in that 
manner), and each had varying results of success on 
large programs. One common thread among 
successful companies was the organization held 
ultimate responsibility for the success or failure of 
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the product and did not rely on a supplier for total 
performance responsibility. In addition, the 
successful organizations had a rigorous method of 
controlling requirement changes and requirements 
creep, which add to the cost, schedule, and 
complexity of a project. Of course NASA need not 
rely solely upon data from outside organizations — 
our own history during the Apollo program 
demonstrates the value of in-house knowledge to 
executing successful operations. 

Finally, there were some relationships between the 
program characteristics within each of the SE&l 
organizational structure types. For example, the Type 
2 SE&I organization has been implemented for two 
major, long-duration, net-centric architectures, with 
multiple critical technologies. There were varying 
program characteristics within each SE&I 
organizational type for most of the other programs. 

In the end, the Ares Projects management team 
elected to work within an organization where 
government maintained integration responsibility, 
with the prime contractor providing SE&I products 
and the government providing technical oversight. 
This structure was deemed the most beneficial for 
executing the Projects because it provides a minimum 
of organizational conflict while maximizing NASA’s 
ability to improve and increase the scope of its SE&I 
activities. 
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